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Abstract 

The changes of learning environments and the advancement of learning theories have increasingly demanded 
for feedback that can describe learning progress trajectories. Effective assessment should be able to evaluate 
how learners acquire knowledge and develop problem solving skills. Additionally, it should identify what issues 
these learners have during the learning processes and why they have these issues. This study depicts visual 
representations of cognitive tasks as crucial points to connect learning and assessment. This study is an 
exploration of these cognitive tasks in complex learning environments and a quantitative representation of these 
measureable objects in cognitive structures. 

Keywords: cognitive task, structural representation, cognitive structures and processes, cognitive assessment, 
Bayesian network, knowledge modeling, measurable objects, knowledge states, medical learning, cognitive 
feature 

1. Introduction 

The assessment of complex learning tasks is a challenge for assessors and researchers. Assessing clinical 
learning of medical students is such a case where the learning tasks are complex, dynamic and without 
pre-arranged scenarios. For example, medical students in their clerkship in a hospital emergency room practice 
how to manage emergency cases under the supervision of a resident or attending doctor. When the medical 
students usually manage patients they cannot anticipate the patients’ symptoms and understand clinical prognosis. 
In such kinds of complex learning situations, the assessors have difficulty evaluating medical students’ 
acquisition of clinical knowledge, development of problem solving skills and situated performances. One critical 
issue is deficiencies of structural representation of clinical tasks. Stated differently, in a complex clinical task, 
chaiiperson, assessor and medical students cannot develop a logical and clear structural representation of the 
learning tasks. 

In this study a professor and a group of medical students discussed deteriorating patient case through a 
video-tape-based instruction. After observing the clinical case all students were interviewed via semi-structured 
interview guide by a group of researchers. The protocol were transcribed and analyzed based on cognitive task 
analysis techniques (Clark & Estes, 1996). A structurally cognitive model was developed and was further 
visually represented by using Bayesian network model (Barber, 2012; Conrady & Jouffee, 2011; Koller & 
Friedman, 2009). Student’s conceptual knowledge and problem solving skills are assessed. Assessment based on 
the structural presentation model can inform a) students to know their strengths and weakness when they manage 
the patient, b) chairperson to understand medical students’ progresses and problems and c) assessors that the 
assessment information is dynamic and diagnostic. 

2. Literature Review 

Effective representations of learning tasks can anchor and structure cognitive behavior (Zhang & Norman, 1994). 
Structural representations can provide learners with cognitive trajectories (Lajoie, 2003) and can allow 
assessment of learning processes (Lu & Zhang, 2013a, 2013b). In other words, an effective representation 
establishes a potential theoretical basis for the assessment. 

The effectiveness of the cognitive task representations is, to a large extent, dominated by the characteristics of the 
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task itself. The cognitive task can be well-structured and can consist of a set of subtasks. These subtasks exhibit 
well-graded knowledge structures (Falmagne & Doignon, 2011). 

However, most cognitive tasks are not well-structured. There are several characteristics about them: (a) there 
are overlaps between subtasks, (b) a cognitive task can be expressed in a latent variable, which is further divided 
into more latent/evidential variables, (c) the presentation from latent to evidential variables is not solely through 
one path, and (d) such a cognitive path is usually explored by cognitive analysis techniques. 

This study includes an examination of the structural representations of the cognitive tasks which are not normally 
well-structured. These tasks are involved in a medical learning activity called “the deteriorating patient.” 
Representations include cognitive structures and processes. Cognitive structures are embedded in learning spaces 
(Falmagne, & Doignon, 2011) and concrete learning events are represented as trajectories within these spaces. 
Cognitive theories can be used to represent concrete learning events in order to understand learning trajectories 
and progress; these theories can also provide evidence for diagnostic assessment. Learning trajectories are 
represented in network structures and learning processes, in Bayesian networks (Darwiche, 2009; Koski & Noble, 
2009; Mittal & Kassim, 2007; Pearl, 1988, 2009). 

From a cognitivist perspective, the acquisition of knowledge and the development of problem solving skills can 
be examined by observing the psychological and behavioral features of human performance (Chipman, 
Schraagen, & Shalin, 2000). Evidence of learning can be elicited from cognitive task performances, which are 
useful in tracing learning and establishing effective assessment (Hollnagel, 2003). 

With recent developments in learning environments (such as web-based and on-line learning systems), complex 
cognitive tasks are coming to dominate learning and assessment processes (Zhang & Frederiksen, 2007; Zhang & 
Lu, 2012). The assessment of learning processes requires information and feedback on the learning trajectories of 
individual learners. Defining, developing and representing learning tasks are crucial for connecting learning and 
assessment. Conventional assessment procedures, such as multiple choice questions, are ineffective because of 
the lack of construct validity. Furthermore, sets of test items randomly selected from banks cannot represent 
expected knowledge structures and conventional test items cannot characterize complex learning task structure 
(Clark, Yates, Early, & Moulton, 2010; Falmagne & Doignon, 2011).This study explores cognitive tasks in a 
complex learning environment, elaborates these tasks into measureable objects, and represents them in cognitive 
structures. 

3. Research Questions 

The critical issue is that assessors and supervisors are not able to properly assess medical students’ learning 
progress because the learning environments and tasks are complicated. We intend to develop structural 
representation as a cognitive assessment model based on the cognitive task analysis. A Bayesian network is used 
to represent the cognitive assessment model. Thus, the following research questions can be addressed: 

1) How is the structural representation model used to represent medical student learning tasks? 

2) How is the Bayesian network used to provide dynamic assessment information? 

3) How are the students’ knowledge and problem solving skills assessed diagnostically? 

4. Theoretical Framework 

Cognitivist perspectives examine the progress of learners in terms of cognitive structures and processes (Zhang & 
Leung, 2007). Acquiring knowledge and developing problem solving skills are accounted for by examining 
structural representations. Also, learning progress can be examined by depicting moment-by-moment changes in 
cognitive processes, and by identifying errors in cognition and performance (Lajoie, 2003). 

4.1 Cognitive Task Representations and Assessment 

Cognitive tasks are usually defined as a series of objects which are implicitly contained in a learning 
environment (Clark & Estes, 1996). These cognitive tasks are “wrapped” into content knowledge of a given 
domain. In a pedagogical context, the instructor and learners rarely place the focus on appropriate cognitive tasks. 
Instructors are able to target some contents, but they do not precisely tailor the content to knowledge 
categorization and different problem solving features. In other words, instructors cannot elaborately design 
cognitive tasks for both (Early, 2007). At the end of a learning period, instructors collect pieces of content 
knowledge arbitrarily or randomly to develop them into test items. Learners’ response to these items can be 
represented at a score level. However, these scores do not report what knowledge these learners acquired, what 
problem solving skills were developed, what kind of errors were made, and what reasons led to these problems. 
No progressive details in the learning process can be recorded and recognized. Thus, an effective representation 
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of a cognitive task in learning is a crucial step towards establishing an assessment that reports diagnostic 
information of knowledge acquisition and skill development. 

4.2 Cognitive Task Analysis and Influencing Factors 

Researchers seek to examine learning tasks from diverse angles. Chipman, Schraagen, and Shalin (2000) 
described learning tasks in terms of a covert-overt continuum. The covert learning tasks, the main focus of 
cognitive tasks in this study, involve unobservable knowledge corresponding to psychological activities, thought 
processes, and goal structures, whereas overt learning tasks involve observable performance. 

Cognitive task analysis is a method for specifying the covert cognitive structures and processes that are 
associated with overt task performance (Clark & Estes, 1996; Gray, 2000; Prasanna, Yang, & King, 2009). 
According to Redding (1992), cognitive task analysis contains three steps: (1) develop visual representations of 
knowledge structures, (2) describe cognitive processes underlying performance, and (3) determine implications 
of results. Olson and Biolsi (1991) argue that cognitive representations are critical for conceptual and 
performance features of cognitive task analysis. Multivariate techniques, such as multidimensional scaling, can 
characterize quantitative features of cognitive representations (Rider & Redding, 1993). Researchers have 
suggested various frameworks, such as representing problem solving skills and strategies (Redding, 1995), 
knowledge structures and measurement modeling (Benysh, Koubek, & Cakvez, 1993), and knowledge modeling 
classification (Essens, Fallesen, McCann, Cannon-Browers, & Dorfel, 1994). 

4.3 Hierarchical Cognitive Task Analysis and Structural Representations 

Hierarchical cognitive task analysis is widely used in the Learning Sciences (Clark, Feldon, van Merrienboer, 
Yates, & Early, 2006, 2007; Merkelback & Schraagen, 1994; Stemler, 2001) and is effective in eliciting and 
representing knowledge and performance. It is often used to examine learning trajectories (Jonassen, Tessmer, & 
Hannum, 1999). Hierarchical cognitive task analysis involves: (1) describing the cognitive tasks and sub-tasks, 
(2) clarifying knowledge (declarative, procedural, strategic, and schematic), and (3) providing a theoretical and 
evidential framework for constructing performance assessments and evaluation. 

5. Methodology 

5. 1 Rationale 

This study uses cognitive task analysis and structural representations to investigate a medical learning task. The 
“deteriorating patient” task requires structural representations to identify the evidential variables needed to 
examine the learning trajectories and individual learner’s progress. Cognitive models are used to characterize 
learning trajectories. Cognitive task analysis delineates the steps from learning tasks to quantitative structural 
representations. 

5.2 From Learning Tasks to Measurable Objects 

The essential parts of structural representation are evidential and explanatory variables. Measureable objects are 
recognized from cognitive tasks. The variables, as the elements of structural representation, are developed from 
measurable objects. The measurable objects function as a “transformation device” from qualitative to 
quantitative cognitive information. 

The notion of measurable objects originated in software engineering as entities with quantitative characteristics 
(Bansiya & Davis, 2002). A similar notion occurs in educational research as physical objects with qualitative 
characteristics (Boulet, 2007). In this study, measurable objects have both quantitative and qualitative 
characteristics. 

5.3 Developing Structural Representations 

A number of steps are involved in using cognitive task analysis to develop structural representations. The 
following steps specify a procedure for applying cognitive task analysis. 

1) Cognitive tasks in a given learning environment. 

The cognitive task in this study involves solving a medical problem in order to stabilize an emergency room 
patient who is deteriorating. 

2) Cognitive framework 

The hypothesis and theories are used in the analysis. In a medical learning situation, students explain how they 
collected patient information. 

3) Cognitive task analysis techniques 
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Knowledge components were recognized at a fine grain level. Protocol analysis of cognitive task analysis is 
used in eliciting knowledge components. 

4) Transform collected information into measureable objects 

Data are coded by rules and raw data are elicited via rubrics to transform information into measurable objects. 

5) Develop evidence variables and explanatory variables 

Variables are defined based on measurable objects. Overt evidential variables are developed from measurable 
objects and covert explanatory variables, from evidential variables. 

6) Structural representation 

After developing evidence variables and explanatory variables, structural representations are assembled based 
on hypotheses and theoretical considerations. 

7) Quantitative model of the structural representations 

Statistical techniques, such as structural equation modeling, multilevel analysis and Bayesian networks can be 
used to change qualitative structural representations into quantitative structural representations. 

8) Knowledge state patterns following dynamic trajectories. 

Certain statistical techniques can be used dynamically to represent learning trajectories or knowledge state 
patterns. The Bayesian network is an effective tool to update learning progress level instantaneously 

6. Analytical Model 

An analytical model was developed based on the procedural steps. 



Figure 1. Procedural steps of cognitive tasks analysis 
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Figure 1 illustrates the relationships in cognitive task analysis components for a quantitative structural 
representation of a medical learning task, in terms of a Bayesian network. 

7. Data Sources in Medical Learning 

7. 1 Participants 

Thirteen third-year medical students volunteered to solve a videotaped clinical case taken from a clinical teaching 
session that was based on a simulated emergency medical scenario, called the “deteriorating patient.” The 10-minute 
video clip shows one learner partially solving the case and a physician and his students dealing with a deteriorating 
patient in an emergency room. The 13 participants were asked to complete the tasks by observing the problem 
solving scenario and were then asked: “If you were the doctor in the emergency room What would you do...?” 

7.2 Pmtocol Cognitive Tasks Analysis 

Participants were required to think aloud. The think aloud protocols require students to say what they would have done 
to solve the problem after watching the video clip. Idea unit analysis (Frederiksen, 1986; Ayala, Yin, Shavelson, & 
Vanides, 2002) is an analytical technique for extracting variables from cognitive tasks. Participants’ think-aloud 
protocols were recorded and transcribed, then the transcribed texts were analyzed (Elm, Potter, Gualtieri, Easters, & 
Roth, 2003; Ayala, Yi, Shavelson, & Vanides, 2002) so that participants’ clinical reasoning processes could be 
identified. Idea units were decided based on theoretical constructs, which can function as criteria for extracting 
evidence variable values and model components for presenting different aspects of competency. Idea units connected 
both evidential data and theoretical constructs. Due to the length of the transcript, it would be too time-consuming 
to analyze the entire text. Two independent raters evaluated a selection representing 25% of the entire total 
protocol. Percentage of agreement on all categories ranged from 78-92%. 

8. Cognitive Models and Bayesian Representation 

8.1 Cognitive Models and Measurable Objects 

Cognitive models can be represented as hierarchical structures and Bayesian networks are appropriate techniques for 
quantitatively representing cognitive models as configurations of evidence and explanatory variables. Evidence 
variables collect evidence from cognitive tasks. Explanatory variables are theoretical constructs representing 
observable judgments from think-aloud protocols. 

8.2 Evidence Variables 

In hierarchical cognitive models, evidence variables are on the bottom layer and are used to extract variable values 
from the data via idea unit analysis. There are ten evidence variables in this study. 

1) Identifying Relevant Information: Students list related information for diagnosis or further actions. Patients may 
manifest many symptoms and focusing on the relevant symptoms is crucial for making decisions, especially when 
time is a factor in life and death situations, e.g.: “He is also hypertensive, he had infarcts before”). 

2) Situation Awareness: This variable is especially important to solve naturalistic problems. Patients may have more 
than one problem and consequently it is important to recognize or express concern for the most pressing problem. A 
patient may have been in a motor vehicle accident and a student might say: “I would wony about him having any 
internal [bleeding].” 

3) Making Judgments: Students express theft opinions and give a diagnosis, for example: “I would think he is 
hypoglycemic.” 

4) Evaluating: Students give a rationale for a judgment, opinion, or action, e.g., “I would also ask for a carotid 
doppler to see if it could be a stroke causing that.. .because he has bilateral carotid bruits.” 

5) Metacognition: It is an advanced cognitive process which assesses and analyzes evaluations and judgments, for 
example: “I’m worried about the pressure, because 1 think he’d be tachycardic; he’d be sweaty, he’d be all of those 
things. But I do not think he’d have high blood pressure.” 

6) Taking Patient History: Students intend to collect more information about patient history. The purpose is to find 
evidence to support current problem solving and reasoning, such as: “I would like to know his meds.” 

7) Physical Examination: Students intend to collect more information through physical examination, e.g.: “I would 
ask for blood pressure on both arms and I would ask for pulses on all four limbs.” 

8) Doing Laboratory Tests: Students intend to collect more information through laboratory tests, such as blood 
sugar. 

9) Giving Medication: This is a cognitive action that the medical student would take after a judgment was made, 
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such as: . .Just give him D50, as quickly as you can.” 

10) Monitoring Patient: This is a more general action that might include management, observation and control of 
patient situation, for example: “Give oxygen... 

8.3 Developing Explanatory’ Variables 

Explanatory variables occupy two layers in the hierarchical cognitive model. It is more convenient to describe the 
explanatory variable from the very bottom level, which is close to the evidence variables. As the model is completed, 
the information of the explanatory variables will be transferred from the top explanatory variable to the evidence 
variables. 

8.4 Upper and Lower Level Explanatory Variables 

As soon as those evidence variables are defined, the lower explanatory ones have to be developed. The lower level 
means that the explanatory variable is closely connected to the evidence one, but they are not observed from the data 
set or they have to be updated by evidence variables. Ten evidence variables have been grouped into four categories. 
The four variables are: Recognizing Information, Deep Cognition, Collecting Patient Data, and Managing the 
Patients. 

Identifying Relevant Information and Situation Awareness share a common feature: Recognizing Information. It is 
obvious that the latter concept indicates the first step in medical problem solving. Making Judgments, Evaluation, 
and Metacognition consist of Deep Cognition, which reflects the second step, following Recognizing Information. 
Taking Patient History, Physical Examination, and Doing Laboratory Tests are composed of Collecting Patient Data 
in the past, present, and future time points. Giving Medication and Monitoring the Patient consist of Managing the 
Patients. Collecting Patient Data and Managing the Patients share a common feature: Cognitive Action. Thus, the 
Cognitive Feature Theory model consists of three sub-models: Recognizing Information, Deep Cognition, and 
Cognition Action; these represent three cognitive process aspects and demonstrate a sequential trajectory. The 
trajectory presents a natural cognitive process as medical students solve a clinical emergency problem. However, 
Cognitive Action cannot be directly observed; it can only be measured by instantiating the variables of Collecting 
Patient Data and Managing the Patients. In brief, the upper level of explanatory variables represents an advanced 
level of clinical cognitive process. 

8.5 Initializing Values in the Cognitive Bayesian Network 

In this cognitive Bayesian network, students’ skills in solving a deteriorating patient problem are expressed as 
both explanatory and evidential variables. Technically these variables are called nodes. The relations between 
variables are connected by the arcs which represent causal relationships. Initializing the estimates of the nodes is 
a necessary step which provides a probabilistic basis for each variable (i.e. node) of the cognitive Bayesian 
network. In other words, the evidential variables, students’ observable behaviors and cognition in solving the 
deteriorating patient problem cannot be updated without the initialized values. 

There are several ways to acquire the estimates of the network nodes, which include educated guesses (based on 
expert beliefs and experience), average values from students’ previous records, and estimates based on student 
performances in similar tasks. Since there are no records from other students in this study, estimates were based 
on expert belief and the literature in the field. In a Bayesian network, the top node is called the parent node and 
the following nodes are called children nodes. We assume that the probability a student successfully 
accomplishes the Cognitive Feature is 0.67; therefore the probability of failure is 0.33. For other nodes 
(variables), we also assume that the probability a student completes a given node is 0.67, otherwise it is 0.33. 
This is also called conditional probability (Koski & Noble, 2009; Korb & Nicholson, 2011). We expect that the 
initialized value of each node is close to the student’s probabilistic level. If the initialized value is not close to the 
expected value, it can be corrected through updating the records until it approaches the expected value (Woolf, 
2009). 

8.6 Brief Description of Entire Cognitive Model 

The entire cognitive model is presented in Figure 2; it consists of three cognitive aspects: Recognizing Information, 
Deep Cognition and Cognitive Action. These aspects represent clinical cognitive processes. This level in the model 
has played an important role in the entire hierarchical cognitive model. 

These three aspects represent a general competency of clinical problem solving, called cognitive feature trajectory 
(CFT) in the graph model. Recognizing Information directly connects to two evidence variables. Deep Cognition is 
composed of three evidence variables. Cognitive Action consists of two sub explanatory variables: Collecting Patient 
Data and Managing the Patient. Collecting Patient Data is updated by three evidence variables; Managing the 
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Patients is updated by two evidence variables. Finally, Recognizing Information, Deep Cognition and Cognitive 
Action are composed of cognitive feature theory in the graph model. 



9. The Bayesian Network Cognitive Model 

There are ten evidence variables which can be used to update the entire cognitive model. A random sampling method 
is applied to test evidence states from one true evidence to ten true evidence observations. The true status means the 
variable acquires a positive value based on the student’s think-aloud text. In order to reduce the tedious process of the 
test, the random process starts from a zero mastery model, which means that the 10 evidence variables do not 
represent any positive values. The student masters zero evidence variables. At the next step, the simulated data of the 
evidence variables demonstrates mastery of one, two, and three evidence variables until it completes the mastery 
model fully; this means the student masters all ten evidence variables. The mastery statuses of problem solving 
processes are listed in Table 1. 

Table 1 demonstrates the relations of various instantiated evidence variables and explanatory variables. The most 
important column is Cognitive Feature, which indicates the general level of the problem solving processes. When only 
one positive evidence is in the model, the Cognitive Feature value is 0.2181 and then increases as the evidence 
increases. When all 10 positive evidence fills in the model, the Cognitive Feature value is 0.7819. The probabilistic 
range is from 0.2181 to 0.7819, which provides a scale to differentiate between various cognitive feature levels. 


Table 1. Updating probabilities of random evidence combination of problem solving process 


Number of Positive 
Evidence 

Cognitive 

Feature 

Recognizing 

Information 

Deep 

Cognition 

Cognitive 

Action 

Collecting Patient 
Data 

Managing the 
Patients 

0 

0.2181 

0.1554 

0.0860 

0.2181 

0.0860 

0.1554 

1 

0.2403 

0.1604 

0.0891 

0.2981 

0.0972 

0.4314 

2 

0.2597 

0.1647 

0.0919 

0.3681 

0.3074 

0.4551 

3 

0.3249 

0.1793 

0.2942 

0.3899 

0.3141 

0.4626 

4 

0.4229 

0.4738 

0.3241 

0.4229 

0.3241 

0.4738 

5 

0.4617 

0.4869 

0.3359 

0.5383 

0.6641 

0.5130 

6 

0.5477 

0.2290 

0.8812 

0.5577 

0.3652 

0.7945 

7 

0.7161 

0.6299 

0.9048 

0.5450 

0.1317 

0.7917 

8 

0.6974 

0.8257 

0.7126 

0.6821 

0.9000 

0.5619 

9 

0.7657 

0.8409 

0.9117 

0.7233 

0.7204 

0.8315 

10 

0.7819 

0.8446 

0.9137 

0.7819 

0.9140 

0.8446 
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10. Students’ Cognitive Feature and Assessment 

10.1 Classification of the Students ’ Cognitive Feature and Measures 

Measures can be defined in different ways. One of the measures is based on the range of the entire probability 
and theory-driven categories. For example, if a researcher believes three categories are appropriate, e.g. lower, 
middle and high, the researcher can define the measure distances as three intervals: (a) 0 < x < 0.33, (b) 0.33 <x 
< 0.67, and (c) 0.67 < x < 1.00. However, the different distributions of given specific data sets are not 
considered in the above measures. If the categories are not sensitive to the given data sets, the categorical 
validity will be weak. 

The second measure can be defined in terms of the practical range of each data set. In other words, we entertain 
the lowest and highest values of a random variable. This measure can effectively describe the classification of a 
set data. If there are n categories, the classification will be defined mathematically, 

X c = R p /n 

Where X c is the interval of the classification, R p is the range of the probability values of a given data, and n is the 
number of classification. We assume the lower, middle, and high are appropriate classifications in our case. So 
here n is equal to 3. 

The third measure can be defined in terms of the practical range of each data set and 

weighted /3s. Regardless of how we consider the weights from any theory and related evidence from data-driven 
reports, a set of weight /3s can be incorporated into the classification. Thus, the classification will be defined 
mathematically, 

X c r Pi R p / n 

Where i is from 1 to n. 

m= i. 

In this study, we have chosen the second measure because this is a data-driven study. We have no theories to refer 
to, but the measure is based on what was discovered from an analysis of the data. Therefore, the second 
measure is an appropriate measure in determination of the classification. 

Students’ problem solving proficiencies are represented by explanatory variables in Table 2, namely Recognizing 
Information, Deep Cognition and Cognitive Action. The probability values represent students’ achievement in 
each explanatory variable. 


Table 2. Students’ cognitive features of problem solving 


Subject number 

Cognitive Feature 

Recognizing Information 

Deep Cognition 

Cognitive Action 

1 

0.5258 

0.7942 

0.7509 

0.7398 

2 

0.4295 

0.4295 

0.5087 

0.5327 

3 

0.4473 

0.4364 

0.7268 

0.5821 

4 

0.4366 

0.4844 

0.5095 

0.6092 

5 

0.4280 

0.5305 

0.5088 

0.6521 

6 

0.4276 

0.5363 

0.5145 

0.6893 

7 

0.4323 

0.5231 

0.4693 

0.6219 

8 

0.4409 

0.7320 

0.5100 

0.5707 

9 

0.4377 

0.7698 

0.5059 

0.6804 

10 

0.4321 

0.5332 

0.4790 

0.6867 

11 

0.4111 

0.5661 

0.7282 

0.7151 

12 

0.4289 

0.7671 

0.5023 

0.6547 

13 

0.4323 

0.7451 

0.5101 

0.6815 
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10.2 Students’ Cognitive Feature Categories 

There were 13 participants in the study. In Table 3, Cognitive Feature presents a general cognitive level in solving a 
deteriorating patient problem based on these students’ data. Recognizing Information, Deep Cognition, and 
Cognitive Action represent three different aspects. The probabilities beneath the four columns of cognitive terms 
quantitatively describe these features. Cognitive Feature can be directly used to represent general “rich” level of 
problem solving processes, while three sub-categories emphasize different aspects. In other words, there might be 
similar values of Cognitive Feature for each subject, but the weights of the three sub-categories are probably 
different. The L, M and H represent lower, middle and high, three skill levels of problem solving processes. The 
values are probabilities that indicate the extent to which the judgment of levels can be trusted. These levels provide 
suggestions to researchers and students and indicate what kinds of information, at what levels, and what is useful in 
a problem solving process, such as clinical information collection and application processes. 


Table 3. Group distribution of subject members in four cognitive categories 


Group 

Cognitive 

Feature 

Recognizing 

Information 

Deep 

Cognition 

Cognitive 

Action 

Subject 

Number 

A 

L 

L 

L 

L 

1,9, 12 

B 

M 

L 

L 

L 

5,6, 7, 10, 11 

C 

M 

L 

M 

L 

4, 8, 13 

D 

M 

M 

M 

L 

2 

E 

M 

M 

H 

L 

3 


Based on the results in Table 3, the Cognitive Feature can be categorized into 2 general groups: M and L group. The 
M group subjects demonstrate that their features applied much richer effective information in clinical cognitive 
process; the L group indicates less information used in the process. The Recognizing Information group represents the 
majority of L level and some M level. Deep cognition category represents all three levels. Cognitive Action expresses 
L level in all groups. 

There are five patterns based on the combination of each explanatory level. Groups D and E present relatively 
intensive information searches, interpretations, and application competencies. We can also observe that the 
categories in Cognitive Feature and in its sub-cognitive categories are not completely identical. In other words, a high 
score in a sub-category does not indicate higher or lower scores whether in general or in other subcategory 
cognitive processes. Thus, the distribution of the values in each column can provide diagnostic information to the 
students. Based on professors and other experts’ experiences, suggestions of how to strengthen student competency 
in collecting effective clinical information might be provided. 

11. Conclusions and Future Work 

This study presents a cognitive task representation in medical learning domains. These cognitive tasks are not 
well-structured, but are embedded in a medical learning context. The learning objectives cannot exhibit explicit 
knowledge structure and states (Falmagne & Doignon, 2011). The authors discovered the effective cognitive 
task representation, which reflected the features of medical students’ learning. The cognitive feature components 
can be referred to further develop well-structured cognitive tasks in similar learning purposes in medical learning 
domains. Such a cognitive representation structure is also used as a data-driven alternative assessment 
framework, such as cognitive diagnostic assessment, performance assessment and dynamic assessment (Lajoie, 
2003). 

This study explores a cognitive process indicating how medical students recognize information, experience deep 
cognition, and take actions in a simulated emergency medical situation. The cognitive process describes students’ 
clinical information collection and application process. A general explanatory variable has been defined as Cognitive 
Feature. The Cognitive Feature consists of three cognitive explanatory variables sequentially: Recognizing 
Information, Deep Cognition, and Cognitive Action. In order to further examine the cognitive feature, a hierarchical 
cognitive model has been established through the analysis ofthink-aloud protocols. The explanatory variables of the 
cognitive feature trajectory cannot be observed. There are potential variables for explaining students’ different 
features in clinical problem solving processes. Ten evidence variables are developed through the idea unit data 
analysis. These instantiated evidence variables are used to update the entire model. 
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A Bayesian network was applied to propagate the information from evidence to explanatory variables. A set of 
simulated network data provides a scale that allows knowing the variation of the students’ levels in clinical 
information collection and application. Cognitive Action results indicate a lower value. Although there are some 
lower and some intermediate values in Recognizing Information, there is large variation of values in Deep Cognition. 
These patterns can provide differential information to analyze different stages in a problem solving process. At the 
first stage in the sequential cognitive feature trajectory, most students focus on the information recognition. They 
have to find and select related information with the clinical case. At the second stage of the cognitive feature 
trajectory, the model presents a variation of patterns, such that five groups fall into three levels in Deep Cognition. 
This stage provides students with more thinking space in making judgments, evaluating, and using metacognition. It 
is challenging for students to make decisions and to take cognitive action in the format of a short video clip. It is 
reasonable that students can contribute more in the cognitive action category if they have more opportunities to 
search for information from available resources, such as electronic clinical libraries or to seek help from the clinical 
tutor in regards to the clinical case scenario. 

In brief, the hierarchical cognitive model, a quantitative cognitive task representation, has effectively identified 
different types of students’ competencies in problem solving processes, based on a verbal protocol analysis. The 
model provides moment-by-moment information differentiation for both students and cognitive feature categories in 
two dimensions. From a student learning dimension, we can observe student cognitive trajectory across these 
cognitive components. Differences between cognitive variables can be examined based on cognitive feature 
components. The cognitive Bayesian network will be more robust in differentiating different student groups and 
cognitive feature categories with updating evidence by the Bayesian network learning. 

The cognitive construct is established by theory-driven analysis. Therefore, the construct validity should be further 
explored with extended clinical data. After all, there are only 13 medical students providing think-aloud data based 
on a short video clip. Exploratory factor analysis and cluster analysis can be employed to examine the data structure 
and construct validity as the number of participants and quantity of data increase. 

The quantitative structural representation of cognitive tasks in complex learning domains provides researchers and 
instructors with a cognitive tool for both learning and assessment purposes. It allows them to observe students’ 
learning behaviors cognitively and assess students’ progress diagnostically following a dynamic path. Based on 
cognitive feature trajectory model in the clinical process and students’ cognitive features of problem solving the 
professor can observe each student’s progress and obtain assessment information at different levels of the 
cognitive categories. 

12. Limitations 

The study was completed with 13 students’ data. The conclusion and suggestions are limitedly generalized to 
different clinical settings and learning environments due to the small sample size. However, the cognitive 
Bayesian representation model can be used as an alternative assessment model in the similar learning 
environments. 
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